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Abstract 


Online education can be leveraged to help current students complete their degrees and to attract new students. While online 
learning is not new, in recent years, it has been moving away from the periphery of higher education and becoming a central 
component of institutional strategies for increasing student enrollment, retention, and completion. Research regarding 
online learning has demonstrated that there is no significant difference between student outcomes in online versus face-to- 
face courses; with some studies showing that online learning can be more effective. Student learning outcomes assessment 
must be part of quality course design and instruction whether a course is delivered face-to-face or online; many best practices 
for face-to-face assessment also apply to online learning assessment. Yet, there are differences between the two modalities that 
must be taken into account when developing online courses. 


The first part of this paper will provide an in-depth discussion of the Open SUNY Course Quality Review Rubric (OSCQR)- 
an online course design rubric and process that is openly licensed for anyone to use and adapt. The aim of the OSCQR 
Rubric and Process is to assist online instructional designers and online faculty to improve the quality and accessibility of 
their online courses. OSCQR also provides a system-wide approach to collect data that informs faculty development and 
supports large scale online course design review and refresh efforts in a systematic and consistent way. This paper then 
explores general considerations of online teaching as they pertain to the assessment of student learning outcomes. Finally, 
specific examples will be given of how online course instructors and distance learning administrators have designed their 
courses and programs to ensure appropriate assessment of learning outcomes. 
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Assessing Student Learning 


in the Online Modality 


Kristyn Muller, Kathleen Gradel, Susan Deane, Michele Forte, Ryan McCabe, 
Alexandra M. Pickett, Rob Piorkowski, Kim Scalzo, 8 Rachel Sullivan 


The number of students enrolled in online courses and degree programs has been 
growing steadily over the past decade. In Fall 2016, 31.6% of all students enrolled at 
U.S. institutions took at least one online course, and over 3 million took all of their 
courses online (Seaman, Allen, & Seaman, 2018). Students generally choose online 
learning because it is more flexible and convenient, especially for those who have work 
responsibilities and/or other life factors that make it difficult to attend class in-person 
(Hannay & Newvine, 2006). Recognizing this need, colleges and universities, both public 
and private, have been increasing their online course and program offerings. 


Online education can be leveraged to help current students complete their degrees and 
to attract new students. While online learning is not new, in recent years, it has been 
moving away from the periphery of higher education and becoming a central component 
of institutional strategies for increasing student enrollment, retention, and completion. In 
a 2015 survey, 77% of chief academic officers at institutions with online offerings agreed 
that online learning is “critical to their long-term strategy” (Allen & Seaman, 2016, p. 5). 
Ina similar survey of chief academic officers from 2017, 83% of the respondents indicated 
that they plan to expand online offerings within the next year (Jaschik & Lederman, 
2018a). Thus, it is important to ensure higher education quality at the course, program, 
and institution levels within the online modality. 


Research regarding online learning has demonstrated that there is no significant difference 
between student outcomes in online versus face-to-face courses (WICHE Cooperative 
for Educational Technologies, 2010). In fact, in some studies, online learning has been 
shown to be more effective (Bernard et al., 2014; Means et al., 2009). However, the 
quality of online education continues to be misperceived, questioned, and debated. In 
a 2015 survey, almost 29% of chief academic officers believed that face-to-face learning 
was superior to online learning (Allen & Seaman, 2016). In a 2018 survey of faculty 
members, only 52% of respondents who have previously taught online and 39% of 
respondents who have never taught online agreed that online courses can achieve the 
same outcomes as face-to-face courses (Jaschik & Lederman, 2018b). Students seem to 
be more optimistic; of those who have previously taken online and face-to-face courses, 
85% believed the online course(s) were either the same or better than their face-to-face 
course(s) (Magda & Aslanian, 2018). Regardless of faculty, staff, and student perceptions, 
however, it is important to ensure that students in online courses and programs have 
learning opportunities that are equivalent to those who are participating face-to-face. 


Online courses can be delivered asynchronously (students participate at different times), 
synchronously (students must be present online at the same time), or a combination 
of the two. Hybrid courses are not fully online because part of the direct instruction is 
delivered face-to-face in a classroom setting. This paper focuses on assessment of fully 
online courses, with an assumption that most of the online activities are occurring 
asynchronously since that is traditionally the dominant instructional method (Johnson 


77% of chief academic 
officers at institutions with 
online offerings agreed 
that online learning is 
“critical to their long-term 
strategy.” 
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& Aragon, 2003). Student learning outcomes assessment must be part of quality course 
design and instruction whether a course is delivered face-to-face or online; many best 
practices for face-to-face assessment also apply to online learning assessment. Yet, there are 
differences between the two modalities that must be taken into account when developing 
online courses. Fortunately, resources, such as course quality review rubrics, are available 
to help faculty members self-assess the design of their course. One popular example is 
the Quality Matters Higher Ed Course Design Rubric, which contains eight standards, 
including Learning Objectives (Competencies) and Assessment and Measurement (QM, 
n.d.). The rubric helps instructional designers and course instructors compare numerous 
aspects of a course with established standards and identify where changes could be 
beneficial. Many colleges and universities use this rubric or other similar rubrics as part 
of their new online course approval and/or continuous improvement processes. While 
these rubrics cannot fully ensure effective online course delivery and/or student outcomes 
assessment, they do provide a framework for instructors to follow in order to make sure 
best practices are followed and student outcomes assessment is intentionally woven into 
each course. 


Open SUNY, the online learning office for the State University of New York (SUNY) 
System, has developed an online course design rubric and process that is openly licensed 
for anyone to use and adapt. The aim of the Open SUNY Course Quality Review 
(OSCQR) Rubric and Process is to assist online instructional designers and online faculty 
to improve the quality and accessibility of their online courses. OSCQR also provides 
a system-wide approach to collect data that informs faculty development and supports 
large scale online course design review and refresh efforts in a systematic and consistent 
way. 


The first part of this paper will explain the Open SUNY Course Quality Review Rubric 
(OSCQR) in more depth, with particular emphasis on the Assessment and Feedback 
section. We will then explore general considerations of online teaching as they pertain to 
the assessment of student learning outcomes. Finally, we will provide specific examples of 
how online course instructors and distance learning administrators have designed their 
courses and programs to ensure appropriate assessment of learning outcomes. 


Open SUNY Course Quality Review Rubric 


Open SUNY developed OSCQR in response to the lack of adoption/use, cost, and the 
prescriptive and evaluative nature of other commercially available rubrics. We needed a 
rubric that would allow greater flexibility for adding and customizing standards, could 
accommodate both new and mature online courses, and could be used in a variety of 
online course quality review models. It was also essential that the rubric and process be 
non-evaluative. Where other rubrics resulted in scores (and potential faculty failure), it 


was important that we view and promote the online course quality review and refresh 
process as a positive professional development experience for online faculty with any level 
of online course development experience. We also wanted a tool that would do more 
than merely point out deficiencies. We wanted the tool to be able to provide significant, 
substantive, and positive suggestions on how to address each standard well. We also 
wanted to incorporate a mechanism to crowd-source additional suggestions and examples 
for each standard, and to rally and recognize our deeply experienced community of online 
practitioners around online course quality assurance activities. 
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The work relied on multi-institutional teams of SUNY online instructional designers, 
librarians, distance learning directors, and technologists. They reviewed and discussed 
the California State University, Chico Rubric for Online Instruction, 20 years of SUNY 
Learning Network (SLN) research-informed best online practices, the SUNY office of 
general counsel’s memorandum on accessibility considerations, and a gap analysis with 
Quality Matters, iNACOUs standards for Quality Online Courses, and Blackboard 
exemplary courses. The resulting rubric was also informed by the Community of Inquiry 
model (Garrison, Anderson, & Archer, 2000), The 7 Principles for Good Practice in 
Undergraduate Education (Chickering & Gamson, 1987), The Adult Learner: A Neglected 
Species (Knowles, 1973), Bloom's Taxonomy (Bloom et al., 1956) and How People 
Learn (Bransford, Brown, & Cocking, 1999). The findings were mapped to the Open 
SUNY fundamental competencies for online teaching. Two components of OSCQR 
were developed: the OSCQR Process, which includes a usage framework and campus 


dashboard, as well as the OSCQR Rubric. 


The OSCQR Rubric has fifty integrated online course design standards and accessibility 
standards, which are organized into six categories: Course Overview & Information, 
Course Technology & Tools, Design & Layout, Content & Activities, Interaction, and 
Assessment & Feedback. The rubric is flexible and designed to be used in a variety of 
course quality assurance approaches: 


¢ By instructors and instructional designers in faculty development and course 
design professional development activities to inform and influence the design of 
new online courses. 

¢ By an individual instructor to self-assess and prioritize design improvements; to 
continuously review, revise, and improve the instructional design of their existing 
online courses. 

¢ By an instructional designer to conduct a formal course review of an online course 
as part of an online course quality review process at the program, department, or 
institutional level. 

¢ As a peer review process, by a team of instructors interested in a peer-review 
model of online course review and continuous improvement (the teams can be 
made up of inter- or intra-disciplinary teams). 

¢ Inacollaborative team model made up of a group of at least 3 people approaching 
the course review process from their own various specialized perspectives, i.e., 
instructional designer, course author, and external reviewers that might include 
other subject matter experts (faculty), online librarian, student, instructional 
technologist, multimedia designer, or other faculty. 


The rubric is unique in many ways: 


e It is not restricted to mature online courses. 

¢ The rubric can be used formatively with new online faculty to help guide, inform 
and influence the design of their new online courses. 

¢ It is non-evaluative. Conceptually the rubric and process approach course review 
and refresh as a professional development exercise, to guide faculty in their 
understanding of improving course design from an effective practice perspective, 
rather than as a course evaluation, or quality assurance procedure. 


The aim of the Open 
SUNY Course Quality 
Review (OSCQR) Rubric 
and Process is to assist 
online instructional 
designers and online 
faculty to improve the 
quality and accessibility of 


their online courses. 
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e The rubric can be customized. Standards can be added, edited, and/or eliminated. 
There is no license fee for use of the rubric. It is shared with a Creative Commons 
license: CC BY-3.0 US. A partnership with the Online Learning Consortium 
(OLC) initiated in November 2016 has resulted in their adoption of OSCQR as 


one of their online quality scorecards. 


Current OSCQR Usage 

The OSCQR rubric and process are currently being used by 56 SUNY campuses and 
800-plus non-SUNY individuals, institutions, and organizations. Implementations vary 
from individual instructors using OSCQR as a self-assessment independently, or in 
conjunction with their campus-based instructional designers, to large-scale institution- 
level online course and program course quality initiatives. These initiatives include the 
incorporation of the standards formatively into new online course design and online 
faculty development activities, summatively for all new online courses prior to their first 
delivery, as well as activities to systematically review and refresh existing mature online 
courses to improve their designs and target accessibility in larger scale institutional online 
course quality initiatives. 


Assessment & Feedback 
The Assessment and Feedback section in the OSCQR rubric identifies the following 


standards: 


44, Course grading policies, including consequences of late submissions, are clearly 
stated in the course information area or syllabus. Learners need to know how their 
work will be assessed in a clear and transparent manner. Grading policies can guide learner 
progress, and promote fair and objective review and assessment of all graded work. Research 
shows that grading policies directly impact learner motivation. Elikai & Schuhmann 
(2010) found that strict grading policies motivated learner learning by associating levels 
of mastery and performance with a specific grade, and guiding achievement progress. 
All assignments and graded activities should have clear goals and criteria for assessment 
within their descriptions. Linking back to grading policies from each graded activity will 
provide more opportunities for learners to understand what is expected from them, and 
the associated guidelines or rubrics can help guide their progress through the assignment 
or graded activity. Including clear course grading policies in both the syllabus and course 
information documents area will also mitigate issues related to learner complaints about 
grades that they have received on assigned work. 


45. Course includes frequent and appropriate methods to assess learners’ mastery of 
content. Consistent and regular assessments help learners demonstrate their progress and 
deficiencies. As learners move through an online course, they should encounter regular 
assignments, activities, and interactions designed to assess how well they have mastered 
the learning content, and how close they are to meeting program, course, or module 
learning objectives. The key to establishing an appropriate assessment strategy is first 
making sure that established goals are measurable, and then mapping activities back to 
those goals to see which best lend themselves to conveying learner mastery. It comes down 


to one simple question—how will you know that learning has taken place? 


46. Criteria for the assessment of a graded assignment are clearly articulated (rubrics, 
exemplary work). Rubrics are recommended as a best practice for communicating criteria 
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and achievement levels for assignments in online courses. According to Wolf & Goodwin 
(2007), rubrics: 


¢ Make learning targets clearer; 

* Guide the design and delivery of instruction; 

¢ Normalize the assessment process; and 

¢ Give learners self- and peer-assessment guidelines. 


Showcasing exemplary work provides learners with a clear example of what outcomes 
the assignment demands, and what mastery levels need to be reached. Before posting 
exemplary work, be sure to get permission from the learner whose work you would like 
to showcase. 


47. Learners have opportunities to review their performance and assess their 
own learning throughout the course (pre-tests, automated self-tests, reflective 
assignments, etc.). Self-assessment involves reviewing one’s own work, determining 
what is good, and detailing what needs improvement. It is a multi-faceted method of 
determining learner mastery, by asking learners to explore their own work, and determine 
a level of performance or mastery. 


Self-assessment plays a role in learner self-efficacy, fosters learners’ abilities to construct 
meaning, and promotes metacognition. By asking learners to check their skill mastery 
levels, or reflect on their own work, they learn to examine their own reasoning and decision- 
making process (Cukusic, Garaca, & Jadric, 2014). In online courses, self-assessments 
provide learners with opportunities to check in to see how they are progressing, and 
often offer opportunities for learners to explore more materials if they still need to master 
concepts or skills. 


48. Learners are informed when a timed response is required. Proper lead time is 
provided to ensure there is an opportunity to prepare an accommodation. All learners 
need clear guidance on when learning activities and assignments are due, and what they 
need to do in order to meet those deadlines. Providing guidance on when timed responses 
are required enables learners to anticipate workload and be better organized (Patterson 
Lorenzetti, 2013; Zimmerman & Kulikowich, 2016). 


49. Learners have easy access to a well-designed and up-to-date gradebook. Learners 
need feedback and guidance to stay on track, especially in online courses. Having a 
course gradebook that is easy to navigate provides learners with the guidance they need 
in order to determine and follow a pathway in their online courses (Schafthauser, 2016). 
Online gradebooks provide instructors with the opportunity to automate, customize, 
and share grades and feedback with learners. Setting up the gradebook with the Learning 
Management System (LMS) should be a core competency of any instructor teaching 
online, as keeping learners on track and informed will promote success and motivation 
in the online space. 


By providing easy access to an up-to-date gradebook, instructors give learners the ability 
to check in on their progress continuously throughout the course term. The added 
functionality and reporting features enable faculty to review and analyze the gradebook, 


re 
The OSCQR rubric and 


process are currently 
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as well as create reports on learner progress and course completion. Learners will also 
benefit by seeing what assignments and other graded activities they have not yet completed. 


50. Learners have multiple opportunities to provide descriptive feedback on course 
design, course content, course experience, and ease of online technology. Learners 
are immersed within the online experience, and can provide useful feedback on how 
courses are designed and delivered. This feedback can be used to guide the efficacy of the 
online teaching and learning process. Providing a channel for feedback, and encouraging 
dialogue among learners can lead to the improvement of ideas and opinions (Mabrito & 
Medley, 2008). For instance: 


¢ Learners may find navigation difficult, or content lacking, which can get in the 
way of successful course completion. Having areas where learners can provide 
feedback to the instructor and/or course designer on navigation, access, and the 
overall learning experience can guide better design choices to support learner 
success while the course is in progress. 

¢ If any new technologies or LMS features are incorporated into the learning 
environment, learners need a channel to report or explain any issues that arise, 
and solutions that they may have found that can be shared with their classmates. 

¢ Independent from end-of-course surveys, channels for feedback on the online 
learning experience empower the learner to have a stake in making the experience 
better for themselves, as well as other learners in the future. These channels also 
enhance group cohesion by exemplifying how instructors value the opinions of 
their learners. 


Online Teaching Strategies 


The standards in the OSCQR Rubric, especially within the Assessment & Feedback 
section, provide a foundation for understanding what must be considered when designing 
a quality online course. However, the actual quality relies on the alignment of course 
content with learning objectives, the use of effective assessment strategies, and the use of 
effective online teaching practices to facilitate online learner engagement and interaction. 
In an online teaching environment, instructors cannot simply duplicate their classroom 
practices, but must rethink and re-conceptualize how to achieve the course learning 
objectives given the unique opportunities and limitations of the online teaching and 
learning environment. Today much is known about how an effective online instructor 
presents content, facilitates discussions, encourages collaboration, and assigns and grades 
assessments. 


Presenting Content 
Modern online courses typically employ a range of tools and formats to aid learners in 


interacting with content, the instructor, and fellow students. A variety of tools can be 
used to make the course content more engaging. Thus, the online modality is a great 
platform for facilitating learner-centered pedagogy (Swan, 2005). Tools like Blackboard 
Collaborate allow students to meet in real-time with their instructors. Students can engage 
through video, audio, or real-time text chat when utilizing this application. Blackboard 
Collaborate works very well for bringing guest speakers into the learning space or for 
virtual office hours. Voicethread, which is a video/audio based discussion platform allows 
students to interact with their instructor and each other by asynchronous video, audio, 
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image sharing, and text. Voicethread can be very beneficial when delivering languages 
online, both World Languages and American Sign Language. Tools like Playposit allow 
instructors to create video based quizzes where students can interact directly with video 
content while instructors check their understanding during various points of the video 
presentation. 


Facilitating Discussions 


Online discussions are frequently used to assess online learning. This approach provids (QQ 


students with a space to make their thinking and learning visible to the instructor and to ; : 
The actual quality relies 


their peers. They can express their points of view, synthesize and critique course concepts/ : 
on the alignment of 


topics, and engage in interactions with their peers. Researchers have studied the efficacy 
of online discussions from both the student and instructor points of view. One research Course content with 

thread notes that online discussion allows all students to engage with course topics, learning objectives, the 
opening up contributions from students who may be reticent to participate in discussions —__yge of effective assessment 
in a traditional face-to-face course (Hall, 2015). Others note that while online discussions 
do have the ability to foster high levels of thinking and learning in a more learner-centered 
format, this is not always the outcome (An, Shin, & Lim, 2009). Many students report 


strategies, and the use of 
effective online teaching 


that their main motivation for participating in online discussions is to receive gradebook Practices to facilitate 


points towards their overall class grade, and some students find discussion board posts online learner engagement 
boring or that the overall experience is isolating (Du & Xu, 2010; Lee & Martin, 2017). and interaction. 


‘There are research-based strategies that instructors can use to ensure that online discussions 
are used effectively for assessment. Many of these strategies involve the format of the 
questions being posed. Other strategies are more focused on the interactions between 
instructors and students. For example, An, Shin, and Lim (2009) studied discussion 
board outcomes across three different sections of the same teacher education course. In 
an analysis of the three course sections, they found that student discussions are limited 
if posting is not mandatory. But, students are much more likely to freely express ideas 
with one another on the discussion board if posts are required and the instructor does 
not respond to students’ initial posts. Mazzolini and Maddison (2007) had a similar 
finding in their content analysis of 40,000 discussion posts across 400 discussion board 
forums. In their study, more frequent posting by the instructor led to fewer and shorter 
posts by students. In survey data, many instructors reported that they asked/posed open- 
ended questions in discussions (a positive intervention), but this self-reported behavior 
was not supported in the content analysis of the discussion board forums. In a case study 
of graduate students enrolled in an online course, Lee & Martin (2017) find that many 
prefer small-group discussion boards, rather than ones that include the entire class. They 
report that this allows them to connect and develop a rapport with classmates. 


Encouraging Collaboration 

An online course done well focuses on access and equity by creating the opportunity for 
every learner to make their thinking and learning visible and open to feedback. The online 
environment is again ideal for this because the only way for a student to be “present” is 
to post something. However, collaboration between students can also occur outside the 
classroom on platforms other than the LMS. Instructors can help facilitate these types 
of interactions by forming small groups or learning teams. It is best to assign students to 
these groups, rather than allowing self-selection, to avoid logistical problems that inhibit 
productivity. If instructors do allow self-selection, they should establish a deadline for 


this process (a week to ten days) and then default to teacher assignment to the groups 
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after the deadline. Small groups are especially helpful for large classes. Small groups can 
develop group presentations, peer review each other’s work, prepare for exams, analyze 
a case study, engage in project-based learning, and more. Just as in well-coordinated 
teaming activities in face-to-face instruction, it is critical that instructors be “present” to 
coach and give feedback (Garrison & Cleveland-Innes, 2005). 


Assigning and Grading Assessments: Formative Assessment 

Research indicates that students and instructors both believe that effective online 
assessments should include a wide variety of assignments—including projects, portfolios, 
self-assessments, timed tests and quizzes, discussion boards, and peer evaluations. 
Feedback on these assignments is also important—and is best when it is meaningful, 
given shortly after the assignment is submitted, and is supported by a rubric (Gaytan & 
McEwen, 2007). Other researchers note that students have to deal with distractions and 
competing obligations while completing coursework. Schacter & Szpunar (2015) note 
that these distractions can be more pronounced in the online environment, where they 
may be tempted to visit other websites or check email while also watching video lectures. 
‘They argue that certain assessment strategies/instructional practices can minimize student 
distractions. For example, Szpunar et al. (2013) found that including memory tests in 
an online lecture helps students pay attention—they report less mind wandering, it 
encourages note taking, and improves learning outcomes when compared to students 
in the same class who watch an online lecture that does not include short memory tests. 


In a well-designed online course, formative assessment is effective because learner 
interaction is central to the process. That interaction gives learners the opportunity to 
express their own “teaching presence” in the learning community with other learners in 
the course and with the instructor. Rather than passive recipients, learners in learner- 
centered online environments become active participants in their own learning, have 
opportunities to interact and collaborate with other learners, to test and socially construct 
their understanding, to bring their own goals and interests to the learning environment, 
to express teaching presence in the course (This notion of Teaching Presence comes from 
the community of inquiry framework authored by Garrison et al., 2000), and to have 
choices about what and how they learn, and how they are assessed—all enhanced and 
facilitated by the power of the internet and the options available in an online learning 
environment. Online learning activities can be designed to ensure that each learner is 
accountable for their own learning, and provide opportunities to assist in the learning 
of others—by questioning others, answering questions, getting feedback, evaluating 
peers, making adjustments in thinking, and having to explain, support, and defend their 
thinking, learners begin to see themselves succeed in their own learning, and adopt an 
identity to match. Self-assessment quizzes are excellent formative assessments, as are 
student metacognitive reflections via online journaling or blogging. 


Assigning and Grading Assessments: Summative Assessment 

There are also opportunities for unique, learner-centered summative assessment 
techniques in the online modality. According to Palloff and Pratt (2013), “A learner- 
centered assessment is an assessment that links what the student is learning in the course 
to the assessment process” (p. 42). Multiple choice tests and quizzes may be easy to grade, 
but writing assignments, collaborative exercises, case studies, and interactive discussions 
provide a more authentic assessment of learner mastery by requiring reflection, synthesis, 
and the creation of new knowledge. Learners can become lost in online courses that fail 
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to measure mastery on a consistent or regular basis, as they have little to motivate their 
participation. Mastering competencies on a regular basis within an online course helps 
learners succeed by developing competence, understanding, and comprehension, which 
leads to the ability to demonstrate competence and elicit feedback (Hulleman et al., 
2010). 


Nevertheless, traditional testing is still common, and perhaps necessary, in many online 

courses. In the past (and even today), some online courses required students to take EE 
tests at specific in-person testing sites, or at remote locations with proctors. Many 
proponents of online education argued that this counteracts or defeats the flexible nature 
of online learning (Khare & Lam, 2008). Today, many online classes have quizzes and recipients, learners in 
tests embedded in the LMS, allowing students to take the assessment wherever they are — learner-centered online 
working remotely. This practice can raise questions related to academic integrity, and  enyironments become 


Rather than passive 


educators and researchers have come to different conclusions on the practice. Some active participants in their 


researchers argue that online examinations align with more constructivist approaches to : : 

‘ ‘ own learning, and begin 
education, because they take away “unnecessary fear” and create a more relaxed learning 
environment for students (Khare & Lam, 2008). In a survey of 110 students who took 
online courses, a majority of students reported taking more time and learning moreinan 1 their own learning, 
online unproctored exam compared to a proctored exam (Myyry & Joutsenvirta, 2015). and adopt an identity to 
O'Reilly & Morgan (1999) also argue that the exam specifications can make it more match. 


difficult for cheating to occur (i.e., asking questions that require synthesizing information, 


to see themselves succeed 


or that must be written in short answer form). A different approach that some schools 


have resorted to is ethics education or honor codes as a means to minimize cheating 
(Khare & Lam, 2008). 


Research on student cheating in online courses is largely based on self-reported data 
from students, where they are asked to indicate if they have cheated on an online exam, 
and if they are more likely to cheat on online exams compared to in-person exams. As a 
whole, findings show that students are more likely to cheat on an online exam (Berkey 
& Halfond, 2015; King, Guyette & Piotrowski, 2009; Watson & Sottile, 2010). Other 
research has compared student outcomes of in-person exams with unproctored online 
exams. Some have not found differences in scores (Ladyshewsky, 2015; Yates & Beaudrie, 
2009), while others have found evidence of cheating (Corrigan-Gibbs, Gupta, Northcutt, 
Cuttrell & Thiess, 2015). Alessio et al. (2017) compared student performance across 
online sections—with half of the sections taking online proctored exams, and the other 
half taking an unproctored online exam. Students who took the unproctored exam scored 
17 points higher and took more time on the exam than those who took the proctored 
version. Some instructors may view this finding through a negative lens—as evidence 
that cheating is likely to occur in an unproctored format. But other instructors may view 
the increased time spent on the assessment positively. Interpretation depends on both the 
learning objectives of the course and the goal/structure of the formative assessment. 


Campus Examples 
The next section provides examples from four different State University of New York 


campuses to explain how course, program, and institution level outcomes can be assessed 
within the online modality. 
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Course Level: Fredonia’s Educational Psychology Course (Multiple Outcomes) 


Context. Fredonia is one of SUNY’s 13 comprehensive campuses, comprised primarily of 
a four-year residential undergraduate population, along with a small graduate enrollment 
of in-service educators returning for the NYS requisite Master’s degree. Located in rural 
Western New York, the campus has been conservative in its approach to online learning; 
there has been a slow, steady growth curve in offering individual online courses across 
majors, but the campus has to date hosted no fully online programs. As such, students— 
at least in the primary academic semesters—tend to parse their experience in online 
learning, often enrolling in few if any online courses; they instead reserve that experience 
for winter and summer (optional) terms. Making a fairly broad generalization, students 
report that—when enrolled in regular semester online courses—they often grapple with 
finding a workable balance between the predominantly face-to-face Fredonia experience 
while engaging in online learning. Coupled with lean campus resources dedicated to 
online learning at Fredonia, students tend to be reliant on the idiosyncratic supports 
afforded by individual online instructors, rather than campus infrastructure. Despite this, 
online courses go through a rigorous review process (now guided by OSCQR), and new 
online faculty complete a required on-demand and face-to-face preparation sequence. As 
part of this experience, assessment is a consistent focus. In addition, the campus has made 
assessment for learning an ongoing priority for all course delivery modalities. 


Learning Outcomes. This example illustrates LEAP domains inclusive of both Jntellectual 
and Practical Skills (i.e., communication, inquiry, critical and creative thinking, teamwork, 
problem-solving) and Jntegrative/Applied Learning. The course is an online version 
of Educational Psychology, required by future educators in our College of Education 
and Communication Sciences majors, and a frequent elective among undergraduate 
Psychology majors. This assignment is guided by five primary student learning outcomes: 


¢ Research evidence-based strategies for proactively building prosocial behavioral 
repertoires among P-12 student learners. 

* Curate and publish multiple products—using functional digital literacy skills— 
capturing current, relevant knowledge and practice. 

¢ Engage in collaborative learning experiences focused on professional expertise. 

¢ Apply understandings to real-life P-12 learning scenarios and simulations. 

¢ Identify personal next steps for extending professional expertise on targeted 
concepts and practices. 


Implementation Steps. This example focuses on positive behavioral support interventions 
and strategies used in the P(preschool) - 12 (Grade 12) school district world. There is so 
much “out there” that is “light”—or common wisdom—paired with substantive research- 


based practice; discriminating “what is what” and how to use it in practice is a perennial 
problem of the education practitioner. The overarching goal of this learning sequence 
is to collaboratively build a class curation of “Fact Sheets” on high-value interventions, 
using materials that students research from reliable sources. Their products become 
“crowdsourced” curations that they then use to solve later course assignments involving 
“intervention challenges” from real classrooms. The process is based on a structured 
sequence of both independent and interdependent work. The steps used by faculty are 
described below; however, targeted outcomes, context, and tools could easily vary. 
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Step 1: The instructor assigns students to initial work teams, generally 
two-person teams. Team membership is posted to a Google Doc (called 
“Intervention Library” in our course), linked to the Learning Management 
System (LMS); this doc is set up giving everyone editing rights. Each team is 
assigned an evidence-based intervention strategy using this same Google Doc. 
Each partner is designated as Team Member A or B. The A/B assignments 
correspond to “job descriptions” delineated in the Fact Sheet template (see 


Step 2). 


Step 2: The instructor builds a shared Google Doc template; the template 
has a series of tables delineating what to do to complete the work. In this 
example, the template is designed with established “job descriptions” for each 
Team member, with nearly mirror images of expectations. This empowers both 
partners to have equivalent but non-redundant accountabilities in generating 
the content from their research. 


Step 3: This shared Google Doc template is used by student teams to make 
their own copies. The (student) document owner makes the template copy, 
then shares it with his/her partner and to “everyone with the link to view;” 
s/he adds the link to the main Google Doc “Intervention Library” doc, and 
moves it into a shared Drive folder set up by the instructor for the full class to 


use. This then makes that original doc a “go to” for everyone in our course to 
access, later on. 


Step 4: Next, team members follow the assignment steps to (a) identify anchor 
resources through their own research, and (b) draft their contributions to their 


own Fact Sheet template. They are expected to use valid research-based resources 
and summarize information in their own words (writing to an audience of 
course peers), while crediting their sources. As part of this work, each team 
member also locates a practical web resource (non-reduplicative of each other), 
as well as an illustrative video vetted for relevance and quality; we call these 
“value-added resources.” 


Step 5: By the deadline, partners then use the commenting (not editing) 
function in their Google Doc, giving explicit feedback to each other on the 
parts that their colleague completed. During this phase, the instructor also 
visits the docs to encourage and to ask questions, to support their mutual 
evaluation and ensure that the collaboration is task-focused and not superficial. 
The commenting tool in Google Docs makes this process virtually seamless, 
while tracking contributions. 


Step 6: After the initial feedback phase, students fine-tune their individual 
work. Using course resources, the partners then decide on the Creative Commons 
License they wish to apply to their work; they add its image, language, and 
a link to the license. At this point, each student inserts the link for his/her 
Fact Sheet doc to the LMS Assignment area, making it easier to input formal 
grades within the LMS. Feedback is provided both within the doc and using 
an assignment rubric, based on both the quality of individual work and the 
collaborative feedback directed to peers. There is a brief time window to (as 
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needed) upgrade work, since everyone will be using these Fact Sheets for next 
work steps. 


Step 7: Next, students are assigned to a new “intervention” team; this team is 
typically comprised of three class members who are different from the original 
research/writing team. Again, using a template, they choose their role; roles 
typically include (a) family member; (b) classroom teacher; and (c) behavioral 
consultant. If a larger team is needed (to balance class numbers), additional 
roles include (a) consultant special educator and (b) building administrator. 
The team then chooses from a variety of real-life “cases” with a focus on 
positive behavioral supports and interventions. These cases vary, drawn from 
(a) actual situations posed by students from their clinical field experiences; (b) 
databases from professional sources such as the JrisCenter (https://iris. peabody. 
vanderbilt.edu); (c) videos (e.g., from The Teaching Channel @ https://www. 


teachingchannel.org; and (d) other professional publications. 


Step 8: Using the same framework of a (different) Google Doc template (see 
earlier steps for copying and sharing, in the Fact Sheet steps), each Intervention 
Team proceeds with (a) identifying the facts of the case; (b) discriminating 
questions pertinent to each role; (c) generating steps for proceeding with 
potential intervention(s); (d) projecting potential obstacles and solutions; 
and (e) designing ways to determine how the intervention’s impacts could be 
measured. Since this is a much more complex process, they are encouraged 
to use small-group Discussion Forums in the LMS, Google Hangouts, the 
“chat” function within the Google Doc, or other online collaboration tools of 
their choice. (Note: They are encouraged to use campus-supported Google and 
LMS tools, making it easy for the instructor to shadow their work, supporting 
them in technical matters, process, and content.) As they work, they complete 
their template with an eye toward both (a) capturing their growing expertise, 
and (b) making their thinking “visible,” since class peers will be accessing their 
work. And, again, there is an intentional feedback phase where team members 
are asked to give substantive feedback to each other, then fine-tune their final 
individual work. This phase ends with the group deciding on their own Creative 
Commons License, as in the “Fact Sheet” phase. The team submits their work for 
summative review and grading. (For detail on the process, please refer to the 
earlier steps comprising the work of building the Fact Sheets.) 


Step 9: Using individual and collaborative learning products and experience, 
students are asked to engage in a summative Discussion Forum in the LMS 
with all class members. The prompt asks students to discriminate how their own 
case analysis both differed and paralleled the work of two other Intervention 
Teams. Follow-up interactions in the Forum ask students to identify concrete 
take-aways from the learning sequence and discussion, citing evidence from 
the curated work and discussions. As with other course Forums, participation 
and contribution are factored into the feedback/grading cycle. The instructor 
takes a guide-on-the side role in this Forum, with the intent of capitalizing on 
students own new expertise; as assistance is needed, the instructor provides it 
by sending email follow-ups to guide students’ public work in the Forum. 
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Step 10: This final step provides individual accountability. Each student 
is asked to generate personal “pinky swears” (a.k.a. professional goals) that 
are personal next steps for extending professional expertise on the targeted 
concepts and practices in the learning sequence. And students are asked to 
identify how they plan to go about taking these next steps. This is published 
to the cumulative “digital portfolio” students compile throughout the course. 
(Optionally, students may elect to add other elements of the sequence to their 


portfolios.) As with other course expectations, this work is factored into the Ds 


feedback/grading cycle. 
Instructors do not 


Lessons Learned. ‘There are several success ingredients relative to this dominate the creation 
collaborative assignment, guided by OSCQR standards. With variation or collaboration 
in targeted learning outcomes and tools, this learning sequence is a fairly process, but are deeply 


sustainable collaborative project that may potentially be replicated across 


ae , vested in shadowing 
diverse disciplines/courses. To summarize: 


and interceding with 


¢ First, this sequence factors in multiple self-, peer-, and instructor- individuals and teams 


(<9 . » . 
guided assessment points, thus providing opportunities for multiple on the side” to provide 
assessment points and measures. An embedded ingredient is using a strategic assistance and 
feedback/grading structure empowering students to be rated for their feedback. 


own work, and not based solely on the final group project or product 
(despite the focus on curated collaborative work flow). (See OSCQR 
Standards 44-49 in the Assessment e& Feedback domain.) 

¢ Second, the experience requires the instructor to engage in both social 
and cognitive presence that is predominantly “coach on the side” 
rather than a “sage on the stage,” capitalizing on peer interactions 
as knowledge is practiced and re-mixed. The instructor does not 
dominate the creation or collaboration process, but is deeply vested in 
shadowing and interceding with individuals and teams “on the side” to 
provide strategic assistance and feedback. The work flow fosters peer- 
to-peer collaboration. (See OSCQR Standards 38-43 in the Interaction 
domain; and Standard 47 in the Assessment e& Feedback domain). 

¢ Athird practice in this learning sequence is the ongoing effort to ensure 
that the product or project has “street credibility’—i.e., assignment 
sustainability and authenticity. This means that it (or some part of it) 
can be used meaningfully for some purpose beyond just getting the 
work done (e.g., as the course proceeds, and perhaps even thinking 
beyond the “walls” of the course itself). As such, this sequence is 
based on iterative, student-driven work; it takes time, and may extend 
beyond the typical “modular” approach to many online courses. (See 
OSCQR Standards 29-31 in the Content e& Activities domain.) 

e Finally, as a “sidebar,” this learning path has capitalized on using— 
and even generating—Open Education resources, to the extent 
possible. Notably, students begin to “walk the walk” by licensing their 
own curated work. (See OSCQR Standards 32-33 in the Content & 


Activities domain.) 
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Course Level: SUNY Empire State College’s Grief and Loss Course (Intercultural 
Knowledge and Competence) 


Context. The following case described the teaching of the course Grief and Loss at SUNY 
Empire State College—one of 64 campuses within the State University of New York 
system. The primary focus of SUNY Empire State is to provide the adult learner—the 
“new traditional student’—flexible paths by which college credentials can be attained. 
These learners are often working full-time and as such require flexibility of asynchronous, 
online environments. Grief and Loss perennially attracts high enrollments in multiple 
sections. The description for this advanced level, 4-credit course reads: 


This course is intended to provide students the opportunity to gain a greater 
knowledge of human grief and the multiple losses individuals experience during 
their lives. You will analyze human grief and loss through exploration of current 
theories of loss and the stages of human grief, the various situations in which 
grief or loss may arise, attending behaviors that can assist in the resolution of 
loss issues, as well as identifying when grief becomes maladaptive and requires 
professional assistance. 


The course appeals to a range of students. Introductory icebreaker discussions reveal that 
students fall along two main trajectories: Those taking it for more personal reasons (hoping 
to better understand a recent loss), and/or those interested in the topic for professional 
reasons (to gain insight and skills into better assisting the populations with which they 
work). SUNY Empire State College offers learners a range of flexible learning modes, 
including independent study, residencies, traditional face-to-face classes, and a multitude 
of fully online programs and courses. Grief and Loss is offered in face-to-face, hybrid, and 
fully online modalities. One of the authors has taught this course both online and face- 
to-face with slight variations in both course content and pedagogy. Where the modalities 
mostly differ is with respect to actual class assignments. One of these assignments is the 
subject of brief analysis here. 


Learning Outcome. This example demonstrates the /ntercultural Knowledge and Competence 
learning outcome, as described in the LEAP framework. Assessing this outcome is didactic 
in nature and heavily relies on learner participation and willingness to critique and engage 
not only course material but also peer learners. Shor and Freire (1987) note use of learning 
activities such as case study critique encourages both learners and teachers to transform 
the learning space into a participatory, collaborative, lived lab space to which student 
perspective is central. This centrality is crucial for learners who aspire to be human service 
practitioners and for whom grief and loss might be an alienating and anxiety-producing 
topic with which they grapple. Doing so in a group setting leverages the power of the 
collective and, as Shor and Freire (1987) suggest, allows teachers to simultaneously draw 
from content expertise while disavowing a “right” to solely claim narrative structure, 


input, and class direction. 


Regardless of modality, intercultural knowledge and competence is integral to stated 
course outcomes. To fully address the complexities of the topic, the course must assess 
a learner’s ability to examine the multiple ways in which diversity of perspective and 
behaviors (including but not limited to gender, culture, religion, and faith traditions) 
impact processes appended to grief and loss. The course and module level outcomes 


specifically supporting this outcome read: 
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1. Examine and analyze the role of rituals, gender, cultural and faith traditions 
in dealing with grief and loss issues. 
2. Compare theories of griefand loss in the context of multicultural perspectives. 


In the face-to-face iteration, these outcomes are measured by in-class discussions, short 
reflection essays, and final projects which ask learners to incorporate considerations of 
diversity into analyses. The online modality demands different pedagogical considerations. 


As such, the course integrates case study discussion to measure and assess this outcome. Ee 
The online modality 


Implementation Steps. Utilizing case studies invites learners into a dialogue about potentially 
real-life scenarios with which they must collectively and individually grapple, interrogating demands different 

the applied distance theory and practice. Both online and face-to-face modalities assess pedagogical considerations 
student understanding of and fluidity with intercultural knowledge via critical analysis of than the face-to-face 

case studies. The face-to-face course was designed to reflect historically consistent student 


iteration. For example, 


feedback noting preference to discuss one case at a time, as an entire class. Feedback from ; 
the course integrates a 


online students, though, notes preference for smaller, focused discussions. We can muse i : 
case study discussion 


as to the reasons for this difference. Shor and Freire (1987) hypothesize that within the 
problem-posing participatory format, both teachers and students have to confront their | tO Measure and assess 
own discomfort with relinquishing of power: Learners are accustomed to what Friere — Intercultural Knowledge 
terms the banking model of education wherein they repeat memorized information back and Competence. 

to “expert instructors” with no real critical engagement of said “facts.” 


The case study assignment asks learners to use prompts to analyze two of five provided 
scenarios which are written to cover a diverse range of both individuals and circumstances. 
The analysis should reference course material as well as relevant, original research. Students 
are also expected to use the same type of references in response to at least two peer posts. 
The prompts and scenarios are: 


¢ Describe and explain what makes this scenario a prolonged loss. Use course 
content to support comments. 

¢ Describe and defend if the person is grieving along a typical trajectory or if 
the grief requires intervention? Use course material to frame and defend your 
choices. 

¢ We discuss attending behaviors in Module 3—what are some attending 
behaviors, questions, verbal or non-verbal skills or cues you would use when 
talking to this person and why? 

¢ Research a few appropriate interventions, and cite sources to support rationale 
for interventions 


Scenarios: 


° Rita, age 52, loss of her job 

° Jon, age 40, loss of a friend 

¢ Liam, young boy, loss of a pet 

¢ Paolo, son, caregiver for mother 
¢ Ruby, 34, loss of partner 
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Lessons Learned. Although the learning activities and reading signal advanced level 
learning, Grief and Loss does not have prerequisites that would limit introductory-level 
students from taking the course. Moreover, the continuum from novice to expert exists 
not only in terms of academic parameters (ability to think critically, engage academic 
work and so forth), but also in terms of learner confidence within the online modality. 
Brooke (2006) notes that online teaching challenges teachers to design strategies to reach 
both novice and seasoned students. Using several smaller, group-based case studies allows 
this instructor to mediate those differences and encourage more confident learners to co- 
facilitate these discrete discussions. Students in the online version of this course report 
being able to keep better track of peer comments when discussion is limited in size and 
scope. 


Student feedback for both modalities aligns with pedagogical suggestions in the OSCQR 
Rubric wherein good online design centralizes student interaction and collaboration 
challenging the passivity of instructor-centric assessments. The ability to create and 
contest social constructions is crucial to this course as learning outcomes (discussed prior) 
value interrogation of social construction of meaning, of “right and wrong,” and of typical 
responses to grief and loss via socio-cultural lens. 


Differential use of case study critique encourages a deeper dive into topics that are often 
uncomfortable for learners to discuss face-to-face. By elucidating the diverse ambiguity 
of socio-cultural determinants of identity, case studies complicate prima facie simplicity 
and invite analyses for which a “right” answer may neither be apparent nor possible. 
Smaller, more curated and facilitated case study discussions afford online learners an 
opportunity to develop and articulate academically rich insights into complex human 
challenges. As the OSCQR Rubric suggests, rote duplication of classroom practices 
within the virtual environment will likely not result in an engaging and efficacious online 
learning environment. Given the learning objectives of this course, differential evaluation 
contexts fetch similar—but modality specific—results. While students in both modalities 
have an opportunity to develop and articulate empathy, the discrete discussion spaces in 
the online course allow students to wrestle with more intimate engagement of provocative 
topics and keep better track of peer comments and scaffolded discussions. 


Program Level: SUNY Delhi’s Orientation to Online BSN (Written Communication) 


Context. SUNY Delhi College of Technology has a unique niche in higher education, 
offering diverse technology-based programs including specialized certificates, associate 
degree programs, baccalaureate programs, and graduate degrees in nursing. The 
online RN to BSN program has been ranked in the top 20 nationwide (Best College 
Reviews, 2019), among the nation’s elite for online baccalaureate education; rankings 


are based upon student engagement, faculty credentials and training, peer reputation, 
student services, and technology. The online RN to BSN program, launched in 2008, 
is committed to academic achievement, civic engagement, and experiential learning, all 
leading to professional advancement. 


Learning Outcome. This example highlights best practices with regard to the Written 


Communication learning outcome at the program level. Graduates of the SUNY Delhi 
RN to BSN program will, in part, upon completion of the program: 
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* use theory, research findings and evidence to support nursing practice; and 
* communicate and collaborate with multi-disciplinary teams. 


Students graduating from Associate Degree programs have minimal exposure to scholarly 
writing. Therefore, the faculty believed that an introduction to Written Communication 
was imperative for incoming students. The course outcomes include: 


As the OSCQR Rubric 
suggests, rote duplication 


Academic integrity with written communication has always been a concern for faculty in of classroom practices 
all academic programs. The online RN to BSN faculty recognize that students entering the — within the virtual 

online RN to BSN program have limited experience with scholarly writing assignments. 
Writing assignments utilizing the American Psychological Association (APA) standards 
are new, confusing, and intimidating to students entering the program. Although many 
believe “plagiarism is plagiarism” no matter what the circumstances, the Delhi RN to BSN 
faculty realized that it is imperative to provide ongoing and scaffolding instruction with learning environment. 


° apply concepts of academic integrity in scholarship ; and 
* integrate APA formatting to scholarly writing and scholarship. 


environment will likely 
not result in an engaging 
and efficacious online 


resources to assist students to achieve the level of writing consistent with baccalaureate 
education and to maintain academic integrity. 


Implementation Steps. All RN to BSN students begin their program with a 3-credit 
course; UNIV 300 Orientation to Online BSN. This course, in part, is an introduction 
to academic integrity to include the online BSN librarian and online BSN writing tutor, 
offering tutorials and ZOOM video-conferencing sessions to assist students in navigating 
scholarly resources and writing in APA format with emphasis on citation and reference 
basics. In addition, there is an entire week that is devoted to discussing the principles 
of academic integrity, the use of Turnitin, how to interpret the results in Turnitin, and 
a Practice Turnitin Submission Exercise. The faculty provide detailed feedback on these 
exercises to students for ongoing improvement. 


Lessons Learned. Faculty recognize that having one course in academic integrity and 
APA format does not transform students into APA scholars. The faculty have instituted 
multiple levels of instruction and evaluation methods for academic integrity in the 
program. Realizing the enormity of the process of students learning scholarly writing in 
APA format, a punitive approach to grading was not consistent with the philosophy of 
the program. The intended outcome for students is to learn APA format as a “work in 
progress” over the course of the program to achieve APA proficiency by graduation. 


Maintaining academic integrity and achieving student proficiency in APA scholarly 
writing in the RN to BSN program, faculty institute processes related to grading rubrics, 
discussions, and assignments. The grading rubrics do not reflect more than 10% of the 
grade based on APA format for discussion questions and assignments. Faculty provide 
detailed feedback using a grading rubric and in-text tracking to demonstrate appropriate 
citations and references with the expectations of improvement in scholarly writing. 
“Turnitin Discussion Checker” portals are posted in each course for each discussion 
question. This Turnitin box is a resource for students to check their discussion posts 
(initial and responses) before posting them, to ensure they are properly cited. This is 
not mandatory or graded however, it is provided as a writing assistance tool that allows 
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students opportunities to improve writing. In addition, for each course, at least one 
writing assignment must be submitted to Turnitin by students. Students do have the 
opportunity to re-submit their assignments to Turnitin after reviewing the report. The 
emphasis on using Turnitin in these assignments is to provide a tool for students to self- 
assess, an opportunity to revise, and ultimately improve writing skills. The goal of assuring 
academic integrity in the RN to BSN program at SUNY Delhi is a scaffolding approach 


to provide students with multiple resources for success. 
Institution Level: FLCC’s Learning Framework (Inquiry and Analysis) 


Context. Finger Lakes Community College (FLCC) made its first foray into online learning 
in the early 2000's. Since that time online enrollment has grown to roughly twenty- 
five percent of the FTE at the institution. FLCC offers 11 degrees and one certificate 
available completely online. FLCC continues to offer students a great online experience 
with supports ranging from 24/7 tutoring, 24/7 helpdesk support, a student concierge, 
student life opportunities for online learners and a lasting connection to the institution. 
With that being said, the core of our mission is student learning and ensuring that the 
students in our online environment receive the same high-quality of instruction as their 
face-to-face counterparts. 


Learning Outcome. In 2015, FLCC adopted a learning framework that assists FLCC 
educators in creating and assessing academic programs and curriculum to preserve 
our unique learning experience while meeting the academic requirements set forth by 
SUNY, The Middle States Commission on Higher Education, and the New York State 
Department of Education. The Learning Framework represents FLCC’s Values (Vitality, 
Inquiry, Perseverance and Interconnectedness). These values are what make the student 
experience unique at FLCC. SUNY and The Middle States Commission on Higher 
Education both require all students to learn certain skills upon completion of their FLCC 
experience. These skills are represented as the beams of the Learning Framework structure. 
The Columns, which support the roof, build on SUNY and Middle States requirements 
by adding general education competencies. The steps to the building connect the students’ 
daily campus learning experiences to the Learning Framework. To better explain each 
aspect of the Learning Framework, please see Figure 1: Learning Framework Graphic 
Organizer. 


As noted, one of FLCC’s values (institutional learning outcomes) is inquiry. The goal of 
the value of inquiry is for students to leave FLCC with the ability to pose insightful and 
productive questions and to be able to generate, evaluate, integrate and cite compelling 
evidence to support reasonable conclusions. ‘This definition of inquiry was distilled from 


the LEAP Inquiry and Analysis Value Rubric. 
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Learning Framework Graphic Organizer 
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Figure I. Learning Framework Graphic Organizer 


Faculty created & defined, these four core 

values are what make FLCC unique. Our institutional 
learning outcomes (ILOs), passed through 
governance in Spring 15, comprise the roof of 

our educational model to symbolize how a student’s 
experience of our curriculum, while gaining support 
from SUNY and Middle States, is sheltered under a 
shared philosophy. 


Lending strength and cohesion to a student’s 
experience of our curriculum, these are the skills 
SUNY and Middle States require us to help all 
students, regardless of program, develop during 
their college experience at FLCC. These outcomes 
were defined by faculty and passed through 
governance in Fall 15. 


Each student passes through our curriculum 
according to the unique requirements of their 
program, fulfilling the specific outcomes that 
satisfy Middle States and SUNY GenEd. 


Defined by faculty and passed through governance 
in Fall 15, we crafted these outcomes so that we 
can use them to measure Middle States and SUNY 
requirements concurrently, thereby streamlining 
program assessment. 


SUNY Gen Ed enables students to acquire 
knowledge and skills deemed useful and 
important for all educated persons, regardless 
of their jobs or professions. 


Required for all bachelor's degree candidates, 
every four-year SUNY campus has a Gen Ed 
curriculum designed to provide a solid academic 
foundation and make transfers within SUNY as 
smooth and seamless as possible. 


Students’ experience at FLCC forms the foundation 
of our understanding of our Learning Framework. 
These outcomes are defined, scaffolded, and 
regularly revisited by program faculty, and 
encompass the specific knowledge, skills, and 
experiences deemed essential for students to 
develop before graduating. 
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Implementation Steps. Our assessment process for general education courses begins with 
the creation of a General Education Assessment Plan (GEAP). As faculty complete the 
GEAP they identify how their course maps to FLCC values. For example, our Introduction 
to Sociology online course maps to the following values; inquiry and vitality. The value of 
inquiry is addressed with students learning the basics of sociological research methods, e.g., 
hypothesis development, data collection and quantitative reasoning. Following this they 
complete the curriculum mapping step where each learning outcome is mapped to a value 
and a cross-cutting skill. To continue with the Introduction to Sociology course, learning 
outcome one is: Students will demonstrate understanding of the methods sociologists use, 
including hypothesis development, operationalization, data collection, and quantitative 
and qualitative analysis. This learning outcome maps to the value of inquiry and the cross- 
cutting skill of critical thinking. Furthermore, the key assessment is identified that will 
ensure that course learning outcomes and institutional learning outcomes are met. Each 
course, regardless of modality is required to be assessed and all instructors are directed 
to require the key assessment of their students. In our online courses the collection of 
this key assessment happens through the Learning Management System and also our 
assessment platform. Materials for lecture and discussion that scaffold the assignment are 
provided to all Introduction to Sociology instructors. Each key assessment is stored in our 
artifact repository using keywords to identify the institutional learning outcomes. On a 
four-year rotation each institutional learning outcome is assessed across all courses that 
have mapped to that outcome to ensure our courses are meeting the institutional learning 
outcomes. During this process a rubric is created to specifically address each institutional 
learning outcome. An example of a 200-level written communication proficiency rubric 
is below. 
200-LEVEL WRITTEN COMMUNICATION PROFICIENCY RUBRIC 


1 2 3 4 
Not Proficient Approaching Proficiency Meeting Proficiency Exceeding Proficiency 
(Benchmark (Milestone) Milestone Capstone) 


Demonstrates a thorough and/or 


Text minimally expresses the 


a0 writer's purpose and/or gives Deore late developing Bemoustiates adeqUsite innovative understanding of 
== itis nakor dion tonidlence awareness of context, audience, consideration of context, Poe lence tenlind ioc] 
z = Readers are confused and/or purpose and to the assigned audience, and purpose and a clear that is responsive to ha task 
Zou F task(s). Work begins to show focus on the assigned task. Work 
Eo insulted and are not persuaded to peerarecclenaliclencaa eiacianareneceiacendioncce and/or responsive to the 
z 2 consider the text's ideas. ercentionsascaninionstand Sercenticnstascumpronetand professional standards of the 
ray - 5 - - » eh 
s Expectations of the audience are expectations. expectations. discipline. Work exceeds 


not met. audience expectations. 


Attempts to use appropriate, 
relevant, and compelling content 
to explore and develop ideas 
within a generally clear 
arrangement of paragraphs, use 
of transitions between 
paragraphs, and organization of 
ideas within paragraphs. The 
organizational logic assists a 
reader's understanding of the 
text's ideas. 


Uses content to develop ideas 
that is inappropriate, irrelevant, 
and/or inaccurate. Arrangement 

of paragraphs, ideas within 
paragraphs, and transitions 
between paragraphs is confusing 
and/or lacks coherence. The 
disorganization makes it difficult 
for a reader to understand the 
text's ideas. 


Uses appropriate, relevant, and Uses appropriate, relevant, and 
compelling content to explore compelling content to explore 
and develop ideas within a logical and develop ideas within a 
arrangement of paragraphs logical, strategic, and relevant 
and/or sections, transitions arrangement of paragraphs, 
between paragraphs, and transitions, and organization of 
organization of ideas within ideas. The organizational logic 
paragraphs. The organizational enhances comprehension and 
logic enhances a reader's conveys the writer's 
understanding of the text's ideas. understanding and authority. 


cTc 
CRITICAL THINKING, 
READING, & WRITING: Content 
Development & Organization 


Demonstrates relevant, accurate, 
and strategic use of credible 
sources to support and 
purposefully extend sound 
reasoning as appropriate to the 
discipline and genre of writing. 


Demonstrates minimal attempts Demonstrates an attempt to use Demonstrates relevant, accurate, 
to use sources, evidence is vague, credible and or relevant sources and strategic use of credible 
and/or reasoning is flawed. to support generally clear sources to support sound 
Sources and evidence establish reasoning as appropriate to the reasoning as appropriate to the 
little authority in the writing. discipline and genre of writing. discipline and genre of writing. 


cTs 
CRITICAL 
THINKING... 
Sources, Evidence 
& Reasoning 


Demonstrates exceptional 
execution of a wide range of 
conventions particular to a 
specific discipline and/or writing 
task(s) including content, voice, 
presentation, formatting and 

stylistic choices. 


Demonstrates consistent use of 
important conventions particular 
to a specific discipline and/or 
writing task(s) including content, 
voice, presentation, formatting 
and stylistic choices. 


Demonstrates inconsistent 
attention to conventions 
particular to a specific discipline 
and/or writing task(s) including 
content, voice, presentation, 
formatting and stylistic choices. 


Attempts to use conventions 
particular to a specific discipline 
and/or writing task(s) including 
content, voice, presentation, 
formatting and stylistic choices. 


CGD 
CONVENTIONS: 
Genre & Discipline 


Text contains some errors in Text contains few errors in 
spelling, grammar, punctuation, spelling, grammar, punctuation, 
or sentence structure. These or sentence structure. Any errors 
errors interfere with do not interfere with 
comprehension at times. comprehension. 


Text contains numerous errors in 
spelling, grammar, punctuation, 
or sentence structure that 
interfere with comprehension. 


Employs language that skillfully 
communicates meaning to 
readers with clarity and fluency, 
and is virtually error-free. 


a 
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Lessons Learned. At FLCC, curriculum and assessment follow the same path, regardless of 
modality. If we do not believe a course can be assessed to meet our standards in a certain 
modality we do not offer the course in that modality. If we were to find that a specific 
online course did not meet our standards of assessment and student learning, we would 
discontinue offering the course in the online format. The alignment between our face-to- 
face assessment process and our online assessment process has proven to be more accessible 
to our faculty and easier for our faculty to navigate. This approach has not diminished our 
ability to offer a variety of courses and programs in the online environment. What is most 
important is that all students, regardless of how they choose to pursue their education, 
receive the same high-quality education at Finger Lakes Community College. 


In Sum 


As online enrollments continue to increase, it is becoming critical for colleges and 
universities to ensure that online students are achieving desired learning outcomes. Using 
the best practices described within the OSCQR rubric, this Occasional Paper provided a 
framework for assessing student learning outcomes in the online modality. Four examples 
from SUNY campuses, at the course, program, and institutional levels were shared to 
highlight a few approaches to consider. Students enrolled in online courses and programs 
should have the same learning opportunities as students in the face-to-face versions, 
although the strategies, assignments, and technologies used may differ. 


What is most important is 
that all students, regardless 
of how they choose to 
pursue their education, 
receive the same high- 
quality education. 
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